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General  Criteria  to  Evaluate  Ill-defined  Problem  Solutions 


One  of  the  USAF  Academy’s  educational  outcomes  is  to  produce  “officers  who  can  frame  and 
resolve  ill-defined  problems.  The  purpose  of  constructing  ill-defined  assessment  tasks  for  our 
students  is  to  assess  the  their  ability  to  recognize  and  contribute  to  the  resolution  of  real  world 
dilemmas  they  are  likely  to  face  in  their  future  careers  as  Air  Force  officers.  Our  first  step  was  to 
define  “ill-defined”  problems; 

Definition 


Ill-defined  problems  are  ambiguous,  interactive  and  ever-changing.  Framing  means  constructing 
a  working  model  and  revising' it  based  on  feedback.  Resolving  means  that  an  ill-defined  problem 
is  never  solved  for  good-  rather  it  is  solved  again  and  again  (re-solved)  as  the  problem  is  framed 
again  and  again;  and,  each  successive  solution  is  more  refined  (resolution). 

In  assessing  student  skills  in  this  area,  it  is  important  to  recognize  that  the  problem  should  be  "ill- 
defined"  from  the  student's  perspective,  not  necessarily  "ill-defined"  from  the  perspective  of 
experts  in  the  field  or  even  the  faculty  member  evaluating  performance.  This  suggests  that 
different  types  of  problems  will  be  appropriate  for  assessing  students  general  ability  and  their 
abilities  within  their  chosen  academic  specialty. 


It  is  also  important  to  point  out  that  it  is  the  solution  process  that  must  be  assessed,  not  just  the 
solution.  In  fact,  a  student  who  had  already  learned  "the  approved  .solution"  from  independent 
reading,  might  be  less  likely  to  demonstrate  a  high  level  of  framing  or  resolving  skills. 

In  addition  to  the  definitions  contained  in  the  outcome  itself,  it  is  important  to  point  out  that  ill- 
defined  problems  have  no  single  absolute  solutions.  However,  solutions  to  these  problems  are 
more  than  a  matter  of  opinion  or  preference;  viable  criteria  exist  for  evaluating  solution  quality. 


Ill-defined  problems  frequently  contain  extraneous  information  may  also  lack  some  necessary 
data.  To  provide  meaningful  assessment,  tasks  to  evaluate  students'  skills  must  be  carefully 
tailored  to  challenge  students  but  not  overwhelm  them.  Assessment  of  both  individual  and  group 
ability  to  frame  and  resolving  ill-defined  problems  should  be  undertaken  across  all  four  academic 
years. 
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LEVELS  OF  PERFORMANCE 

Framing  and  Resolving  Ill-defined  Problems 


EXCELLENT 

-  identifies  most  important  ill-defined  aspects  of  problem  as  well  as  general  "ill-defined"  problem 
-  nature 

-  uses  goal,  mission  or  other  ultimates  to  structure  problem  space 

-  keenly  aware  of  personal  perspective  and  bias  and  compensate  effectively;  also  aware  of  larger 

contexts 

-  use  general  principles  and  ^damental  concepts  to  frame  overall  problem  space  and  as  solution 

tools 

-  systematically  works  through  problem;  often  makes  multiple  passes  through  the  problem  space 
.  as  conditions  change  in  order  to  assesses  consequences  of  changes  or  alternatives 

-  unsuccessful  attempts  regularly  used  to  better  understand  problem  and  solution  process 

-  generates  rich  variety  of  altematives,  tests  them  objectively  arid  selects  rationally 

-  appropriate  level  of  confidence  and  commitment  to  eventual  solution 

SATISFACTORY 

-  aware  of  general  "ill-defined"  nature  of  the  problem  and  some  of  the  specific  problem 

deficiencies 

-  may  structure  problem  space  based  on  superficial  problem  characteristics  or  unwarranted 

assumptions 

-  sornewhat  aware  of  personal  perspective;  evidence  of  awareness  found  throughout  solution 

process  but  some  important  connections  to  more  general  contexts  not  understood 

-  tendency  to  use  particular  tools  and  mechanisms  appropriately  but  may  lack  ability  to  modify 

and  adapt  them  appropriately  due  to  incomplete  understanding  of  underlying  principles. 

-  works  through  problem  systematically  but  may  omit  necessary  reconsideration  of  assumptions 

-  unsuccessful  attempts  recognized  and  abandoned 

-  generates  multiple  potential  solutions  but  may  not  fully  consider  all  of  them  or  use  appropriate 

criteria  to  select 

-  likely,  to  lack  confidence  in  solution;  limited  commitment  without  encouragement  or  support 

DEFICIENT 

-  unaware  of  either  general  or  specific  characteristics  that  preclude  routine  solution  procedures 

-  apparently  unaware  of  personal  perspective  or  bias;  assumes  single  perspective  is  sufficient 

-  random  or  inappropriate  application  of  tools;  may  not  be  able  to  provide  reasons  for  approach 

selected  . 

-  imsuccessfiil,  sporadic,  apparently  random,  attempts  at  problem  lead  to  frustration  and 

abandonment 

-  unsuccessful  attempts  based  on  imtenable  assumptions  not  recognized. 

-  fully  commits  to  first  apparent  solution  path  and  follows  it  through  to  completion  without 

reconsideration 

-  likely  to  display  either  no  confidence  in  solution  or  process  (may  claim  problem  is  impossible) 

or  be  inappropriately  confident  and  overly  committed  to  obviously  ineffective  solution 
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Assessment  Problem  -  Aircraft  Deployment 


KC-135  Deployment  Problem 

SS  AN _ _ 

,  The  commander  has  asked  you  to  put  together  a  maintenance  support  force  for  the 
deployment  of  three  KC-135s  to  a  temporary  operating  location.  When  we  recently  deployed 
three  aircraft,  our  support  force  had  30  maintenance  personnel  and  we  were  able  to  fly  a  total  of 
36  missions  for  the  two  days  we  were  deployed.  Two  years  ago,  we  took  16  personnel  and  five 
aircraft  to  Eglin  AFB,  FL  where  we -flew  40  missions  in  the  four  days  we  were  deployed.  Our 
sister  squadron  just  returned  from  a  five  day  trip,  where  they  flew  ten  missions  in  the  five  days 

they  were  deployed,  using  jpst  five  technicians  and  one  aircraft. 

•/ 

The  commander  wants  to  fly  30  missions  in  the  three  days  we  are  deployed.  You  plan  a 
maintenance  support  force  that  includes  20  personnel.  • 

How  effective  do  you  think  we  will  be  during  the  deployment,  as  measures  against  the 
commander’s  goal  of  30  missions?  Support  your  position  by  describing  how  you  arrived  at 
your  answer. 


Questions  on  page  2: 

-  On  a  scale  of  one  to  ten,  how  confident  are  you  that  your  answer  is  correct? 

-  On  what  do  you  b^e  your  level  of  confidence? 

-  Does  your  answer  depend  on  any  particular  assumptions?  What  are  some  of  the  important 
ones? 

-  If  you  could  choose  to  have  one  more  piece  of  information,  what  would  that  be? 

-  How  would  that  additional  information  change  your  original  answer? 
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Assessment  Problem  -  Astro  Ideas 


You  are  assigned  to  make  recommendations  to  the  Head  of  the  interplanetary  exploration  space 
mission  group  at  NASA.  The  subject  mission  is  to  Mars  in  12  years. 

-  Your  range  of  options  include:  machine  only,  crewed,  or  a  composite. 

-  You  may  examine  a  tradeoff  for  a  rover  or  extra  fuel  for  vehicle  hopping. 

-  You  may  also  examine  a  tradeoffs  for  the  trajectory  for  the  transfer  orbit  to  go  from  Earth  to 
Mars  and  back:  direct,  Hohmannj  Or  0TB  (one  tangent  bum). 

-  Your  main  concern  once  you  decide  on  whether  cr  a  crew  should  be  on  the  space  vehicle  is 
as  follows: 

1)  Machine  only  Option- What  instmments  must  be  included?  Should  there  be  robots? 
Should  there  be  a  sample  and  return  or  on  Mars  processing  only?  Any  other 
iniportant  questipns  you  seem  to  think  are  important. 

2)  Crewed  Option-How  many?  Backgrounds?  Age?  Abilities?  Gender?  Personality 
type?  International  or  US  only?  Any  other  important  questions  you  seem  to  think  are 
important. 

3)  Composite  Option-How  many  machines  do  you  forgo  to  add  people?  Also  all  the 
above  apply. 

-  In  the  past,  only  male  crews  went  to  the  Moon.  The  crews  consisted  of  two  individuals  in  a 
lander  and  one  remaining  on  the  main  vehicle. 

-  Mercury  and  Gemini  programs  started  out  with  one  and  two  individuals  before  progressing  to 
three  in  the  Apollo  program. 

-  The  SkyLab,  Shuttle,  and  Mir  programs  included  a  varied  crew  depending  on  mission  duration 
and  objective. 

What  recommendation  do  you  make  for  the  mission?  Support  your  position  by 
describing  how  you  arrived  at  your  answer. 


Questions  on  page  2: 

-  On  a  scale  of  one  to  ten,  how  coiifident  are  you  that  your  answer  is  correct? 

-  On  what  do  you  base  your  level  of  confidence? 

-  Does  your  answer  depend  on  any  particular  assumptions?  What  are  some  of  the  important 
ones? 

-  If  you  could  choose  to  have  one  more  piece  of  information,  what  would  that  be? 

-  How  would  that  additional  information  change  your  original  answer? 
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The  following  problem  is  purposely  ill-defined,  vague,  and  ambiguous.  Insufficient  as  well  as 
extraneous  information  may  exist  in  the  givens,  and  there  may  be  several  solutions  at  the  outset 
that  would  have  to  be  refined  after  obtaining  more  information.  Our  purpose  here  is  to  evaluate 
your  ability  to  frame  the  problem,  given  only  the  information  presented,  and  discuss 
assumptions,  “firee”  parameters,  possible  solutions,  and  tradeoffs. 

Background  Information 

Figures  1-4  depict  graphical  representations  of  the  general  magnitude  specifications  for  the  four 
conrunon  types  of  filters:  lowpass,  highpass,  bandpass,  and  bandreject  (notch).  In  this  problem, 
you  will  be  specifying  a  filter  type  (or  types)  and  determining  as  many  of  the  appropriate 
parameters  as  possible.  , 


Figure  1 .  LOw  Pass  Filter  Figure  2.  High  Pass  Filter 


Figures.  Band  Pass  Filter  Figured.  Band  Reject  Filter 


where 


Ap  => 

K  => 

fs  => 

fc  => 


the  absolute  value  of  the  passband  attenuation,  in  dB 
the  absolute  value  of  the  stopband  attenuation,  in  dB 
the  lower  end  of  the  stop  band 
the  cutoff  frequency 

stopband  1  ixnivS  i-Ojk  O' cr  bandreject  filters 
passband  limits  for  bandpass  or  bandreject  filters 
bandwidth  of  bandpass  of  band  reject  filters 
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We  are  more  interested  in  the  process  you  go  through  than  we  are  in  a  bunch  of  equations,  so 
please  give  us  all  your  thoughts  as  you  go  along. 

Here’s  the  problem: 

You  are  a  Lockheed  electrical  engineer  working  on  a  digital  data  transmission  problem.  You 
have  asked  an  engineer  at  another  location  to  specify  the  problem  as  clearly  as  possible  by  email 
so  you  can  make  a  presentation  to  your  boss  by  3:00  p.m.  this  afternoon.  Unknown  to  you,  the 
engineer  at  the  other  end  is  wearing  a  Hawaiian  shirt  and  is  going  to  jump  in  the  car  and  head  for 
the  airport  as  soon  as  he  sends  you  the  email.  Unfortunately,  nobody  else  is  familiar  with  the 
problem  he’s  having  so  you’re  stuck  making  the  best  of  it  in  your  presentation  to  the  boss. 

Here’s  the  email:  ' 

Hi 


Thanks  for  taking  a  look  at  this.  We  have  a  coaxial  line  coming  into  the 
cockpit  of  the  C-17  that  carries  a  single  TTL-compatible  RPM  signal  to  a 
receiver  behind  the  cockpit  instrumentation.  Unfortunately,  the  coax  is  routed 
-pretty  close  to  a  400-Hz,  AC  generator  that  is  playing  heck  with  our  signal.  I’ve 
got  a  bandwidth  on'the  digital  signal  of  about  1  kHz  to  50  kHz. 

I  asked  them  to  reroute  the  coax  and  they  said  it  can’t  be  done  without  adding 
about  $10K  to  the  instrumentation  package  price.  (I’m  already  in  hot  water  as  it 
is,  and  don’t  want  to  add  any  more  expense  than  I  have  to!)  I  know  you  probably 
need  to  have  a  DC  component  figure,  but  our  spectrum  analyzer  is  broken  and  I 
really  don’t  have  time  to  get  a  worst  case  duty  cycle  on  the  digital  signal  because 
I’ll  be  late  for  my  plane  to  Hawaii!  So,  is  there  some  kind  of  a  cheap  filter  or 
something  we  can  use? 

Thanks  again.  Good  luck! 

Your  ex-friend.  Bob 


P.S.  I’ll  bring  you  back  a  can  of  macadamia  nuts. 


Your  Task 

For  your  presentation  to  the  boss,  sketch  a  filter  specification  (or  specifications)  and  identify  as 
many  parameters  as  you  can  of  those  given  on  the  previous  page.  Discuss  with  your  boss  any 
“free”  parameters,  as  you  see  them,  and  how  you  could  choose  those  parameters  to  optimize 
cost,  oerformance,  etc.  Specify  all  assumptions  you  must  make  for  each  choice  of  solution. 

Tell  the  boss  what  information  you  would  really  need  to  completely  specify  the  filter  (or  filters). 
Good  luck! 
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Questionnaires  and  Survey  Development:  Principles  of  Good  Practice 


GoaWABET  Criterion  (Criteria'^  Addressed 

This  session  will  present.an  introduction  to  good  practices  in  the  development  of  surveys 
designed  to  assess  the  new  ABET  criteria.  Principles  to  follow  wiien  designing  a  survey 
or  questionnaire  to  be  used  in  collecting  data  on  student  outcomes  will  be  presented 

Presentation  Format 

/ 

The  session  will  include^several  brief  information-dense  lecture  periods  each  followed  by 
an  activity  designed  to  help  learn  the  pesented  material.  Particip^ts  will  work  in  small 
groups  on  these  activities.  Attendees  should  have  paper  and  pencil  for  idea  development 

Session  Summary 

Principles  to  follow  vdien  designing  a  questionnaire  or  survey  to  be  used  in  collecting 
data  on  student  outcomes  will  be  presented.  Issues  to  be  hig^ghted  are:  critical  questions 
which  need  to  be  answered  prior  to  instrument  development,  common  mistakes  to  avoid, 
instrument  structure  and  format,  general  principles  of  sampling,  and  on-line  and  Internet- 
based  data  collection. 

Topics  with  activities  will  include  developing  a  sampling  design,  development  of  mail 
survey,  telephone,  and  interview  instruments,  and  development  of  attitude  questionnaires 
and  observation  protocols. 

Key  Words 

Survey  sampling,  mail  surveys,  telephone  surveys,  attitude  questionnaires 
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ATTITUDE  SCALING 


I.  Thurstone  scaling 

A  Selecting  single  concept,  idea,  or  construct  for  scaling 
eg.  War,  marriage,  abortion,  mathematics 

B.  Collection  of  statemfents  about  the  concept:  non-factual,  opinion-oriented; 
select  about  80-100  for  analysis, 

.  eg,  I  like  arithmetic  most  of  the  time. 

Abortions  should  never  be  performed  under  any  circumstances. 

War  is  usually  a  good  thing,  everything  considered, 

C.  Placement  of  statement  along  1 1  point  continuum  from  (-)  1=  most  negative  statement 

to  (+)  1 1=  most  positive  statement,  with  6=  neutral  or  nonjudgmental  statement  . 

-  use  50-200  subjects  to  do  placement 

-  evaluate  distribution  of  each  statement: 

Median; 

for  example,  statement:  Abortions  should  never  be  performed,,, 

1  2  3  4  5  6.,. 

n:  150  50  0  0  0  0 

I 

%ile  50 
score  1.17 

for  example,  statment:  Abortions  should  be  performed  only  to  save  the  life  of  the 


mother. 

1 

2 

3 

4  5 

n:  10 

40 

90 

1 

40  20 

%ile 

1 

50 

score 

3.05 
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Variabffity; 

-eliminate  itenis  with  ranges  greater  tiian  6  or  7 
-  examine  conditional  di^butions  of  adjacent  or  close  items: 


have 

have 


♦  give  items  to  200-300  respondents  to  endorse  each  statement  (+)  agree 
or  (-)  disagree 

♦  examine  joint  endorsements  of  one  item  (a)  with  another  (b), 
using  ^  index  such  as 

♦  the  distribution  of  the  J[b  *s  should  dcrease  around  item  a  on  eitiier 
side  of  its  scale  value;  t^  is,  items  with  similar  scale  val\^  should 

a  high  similarity  index,  vhile  items  further  away  on  the  scale  should 

scale  values  that  drop  away  with  distance.  Throw  out  items  with  poor 
characteristics. 
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SAMPLING  PRINCIPLES 

(adapted  from  Kish,  L.  Survey  Saa^ling.  Wiley,  1965) 

SAMPLING  FEATURE  SIMPLEST  VERSION  COMPLEX  VERSION 


Probability  Equal  probability  Unequal  prob- 

for  selectibn  for  all  persons  abilities  due 

to  study  con- 

/  ditions 

Sampling  ‘  Sample  each  person  Sample  first 

clusters  of 
persons,  then 
persons 

Strata  Sample  from  entire  Separate  sample 

population  for  various 

subgroups  such 
as  males/female 

#  of  Phases  One  sampling  time  Samples  drawn 

from  initial 
sample  that 
is  used  to  get 
information 
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INTERVIEW  PROCEDURES 

1.  Structure;  Unstructured 


Partial  Structure 

Semistructured 


Totally  structured 

2.  When  to  use 


3 .  Styles  of  interviews 

Nondirective 

Focused 


Multiple  interviewers 
Multiple  interviewees 

4.  Potential  interviewer  problems 


5.  Telephone  interviews 


6.  Tips  for  interviewing 
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SURVEY  QUESTIONNAIRE  DEVELOPMENT 

(adapted  from  Krathwohl,  D.R.  (1993).  Methods  of  Educational  and  Social  Science 
Research.  New  York:  Longman. 

Major  Issues 

1 .  What  to  ask 

2 .  How  to  ask  it 

3.  Tips  on  Question  construction 

a.  Understandable  language 

b.  Check  interpretation 

c.  Keep  as  short,  simple  as  possible 

d.  One  concept/issue  per  question 

e.  Avoid  biased  phrasing 

f.  Framing  the  question 

g.  Avoid  potential  embarrassment  in  responses 

h.  Help  negative  responses  by  beginning  positively 

i.  Maintain  impersonal  orientation 

j.  Avoid  negative  questions  where  possible 

k.  Develop  best  range  for  multiple  choice  options 

l.  Don’t  focus  or  bias  responses  by  order  of  earlier  questions 

m. 

4  .  Ordering  the  Questions 

5.  Format 
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OPTIONAL  EXTENDED  YEAR  PROGRAM  EVALUATION 
STUDENT  SAMPLING  PLAN  1996-97  Year 


L  Characterizing  0£YP  Districts 

School  district  reports  from  TEA  were  the  basis  for  describing  the  number  of 
students  enrolled  in  the  OEYP  program  for  the  1996-97  school  year.  A  total  of  492 
districts  provided  reported  student  enrollment.  The  distribution  of  number  of  students 
enrolled  was  the  basis  for;  sampling 

Student  enroUmeht  varied  from  2  students  to  almost  16,000.  TTie  empirical 
distribution  was  reviewed  and  provided  meaningful  breaks  as  follows: 


District 

Number 

Total# 

%  districts 

%  students 

#  stu¬ 

Enrollment 

of  Districts 

Enrolled 

Selected 

Selected 

dents 

Selected 

Very  Large 
(575-16,000) 

50 

100,629 

100% 

10% 

10,000 

Moderately  Large  73 
(251-750) 

23,580 

25% 

50% 

3,000 

Medium  Size 
(100-250) 

123 

15,602 

25% 

100% 

4,000 

Small 

(31-99) 

128 

5,969 

25% 

100% 

1,500 

Very  Small 

118 

1,947 

75% 

100% 

1,500 

Totals 

492 

147,727 

45% 

(221) 

13.5% 

20,000 

n.  Sampling  Procedure 

Within  each  district  size  category  above,  all  reporting  districts  were  given  equal 
weight.  While  the  original  plan  was  to  sample  the  28  districts  that  participated  all  four 
years,  only  one  of  the  districts  provided  student  lists  for  the  1995-96  school  year,  so  that 
stratum  was  abandoned.  Districts  were  selected  using  random  numbers.  If  a  district  did 
not  provide  student  data,  it  was  replaced  by  the  district  immediately  below  it  in 
alphabetical  order. 
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On-site  Interview  (Administrator)  Date _ Interviewer _ _ 

District  name _ _  District  # _ Location  of  interview _ 

Name  of  person  interviewed _ Current 

position _ : _ ^ _ _ 

1)  What  was  your  involvement  with  OEYP  in  the  District? 

2)  What  years  were  you  involved  with  OEYP? _ ^93/04 _ ^94/95 _ 95/96  _ 96J91 

3)  What  is  the  current  status  of  OEYP  in  the  district?. 


Is  OEYP  being  implemented  in  the  1 998/99  school  year  Yes _ No 

or  summer  1998?  Yes _ No 

Comment: _ : _ _ 

4)  How  was  OEYT  implemented  in  the  district? 

^  Last  year  it  was  Implemented:  Previous  years,  If  different: 

a)  Extended  format: _ ^Day 

_ Week 

—Year 

Typical  class  sizes  per  teacher: 

What  was  the  number  of  days 
offered  in  OEYP? 

Last  year  It  was  Implemented:  Previous  years,  If  different: 

What  was  the  length  of  the  _  _ 

instructional  day? 

b)  Were  centralized  campus(es)  used  or  did 
children  attend  their  regular  school? 


_ ^Day 

—Week 

—Year 


c)  How  was  transportation  and  breakfast/lunch  handled? 


d)  Were  any  private  sector  services  used  in  instruction  or  support? 


e)  Was  there  a  parental  involvement  component? 
_ Yes  No 


_ Yes _ ^No 
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Yes  No 


e)  Was  there  a  parental  involvement  component? 

_ ^Yes _ ^No 

How  was  it  implemented? 


Last  year  it  was  implemented:  Previous  years,  if  different: 

f)  Was  there  a  professional  development  component  for  OEYP  teachers?  How  was  it 
implemented?  , 


5)  What  were  important  aspects  of  the  program  focus  and  implementation  in  the  last  year  of 
implementation? 


How  did  these  differ  from  previous  years? 


6)  What  were  the  greatest  problems  in  operating  OEYP? 


7)  What  benefits  did  the  District  get  from 

OEYP?__ _ _ 

Benefits  for  students? _ 

8)  Is  the  District  continuing  OEYP?  Yes _ No 

Why  or  why  not?  _ 


Is  the  pistrict  eligible  for  funding  in  1998-99?  _ ^Yes  _ No 

9)  Do  you  think  the  OEYP  program  was  effective? _ Yes  _ ^No 

For  students?  _ Yes  _ ^No 

Would  you  support  continuing  the  OEYP  program?  With  or  without  State  support?  Why  or  why 
not? 
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Assessing  the  Effectiveness  of  the  First  Two  Years’ 
Mathematics  Curriculum  in  an  Engineering  School 


Session  Summary; 

Colorado  School  of  Mines  (CSM)  recently  instituted  a  major  curriculum  revision.  The 
process  started  with  the  Academic  Planning  Committee’s  formulation  of  the  “Profile  of 
the  Colorado  School  of  Mines’  Graduate.”  The  Curriculum  Reform  Steering 
Committee  (CRSC)  was  formed  to  guide  the  planning  and  implementation  of  the 
revisions  necessary  to  comply  with  the  “Profile.”  Sub-committees  were  formed  to 
examine  our  curriculum  ahd  its  fit  with  that  document,  to  study  what  comparable 
institutions  were  teaching,  and  to  determine  the  changes  we  needed  to  make.  The  CRSC 
developed  a  curriculum' framework  that  was  adopted  by  the  faculty.  This  paper  is 
concerned  with  the  formulation  of  the  first  two  years’  mathematics  courses. 

Needs  Assessment;  The  Mathematics  and  Basic  Sciences  Sub-committee  (MBS),  made 
up  of  faculty  from  all  of  the  departments,  discussed  at  length  what  requirements  were 
needed  in  mathematics,  physics,  and  chemistry.  The  committee  recommended  that  the 
calculus/differential  equations  sequence  go  from  a  4-3-3-3  credit  hour  sequence  to  a  4-4- 
4-4  one.  They  developed  lists  of  topics  to  be  included  in  each  course  and  examined  how 
those  topics  matched  with  the  comparable  science  classes.  Departments  were  asked  to 
provide  input  for  the  new  courses  in  mathematics  and  the  basic  sciences.’  Several  Faculty 
Forums  were  held  to  discuss  the  framework  and  the  content  of  courses. 

Planning  New  Courses;  Students  at  CSM  traditionally  take  the  second  semester  of 
calculus  and  the  first  semester  of  physics  during  the  spring  semester  of  their  freshman 
year.  For  physics,  the  students  need  vectors.  For  physical  chemistry,  the  students  need 
partial  differentiation.  As  many  of  the  traditional  second  semester  calculus  topics,  such 
as  techniques  of  integration,  have  become  less  essential  in  this  era  of  symbolic 
manipulators,  these  were  eliminated  in  the  new  courses.  The  new  Calculus  for  Scientists 
and  Engineers  II  consists  of  vectors,  lines,  curves,  planes  in  space,  partial  differentiation, 
and  multiple  integration.  The  third  semester  course  is  surface  and  line  integrals. 

Green’s  and  Stokes’  theorem,  sequences  and  series,  and  techniques  of  integration  as  they 
apply  to  the  solution  of  differential  equations.  The  third  semester  topics  support  the 
Physics  II  course  and  the  second  course  in  Physical  Chemistry.  All  calculus  courses  have 
a  strong  problem-solving  component.  The  Undergraduate  Committee  of  the 
Mathematical  and  Computer  Sciences  Department  took  the  input  froih  MBS  and  the  other 
departments  and  developed  the  performance  objectives  and  syllabi  for  the  new  coiuses. 

Development  of  Problems  that  Reflect  the  Language  of  Science  and  Engineering; 
Engineering  and  science  faculty  complained  that  students  finished  their  calculus  sequence 
unprepared  or  under-prepared  to  do  the  mathematics  in  their  courses.  They  commented 
that  students  had  difficulty  translating  the  calculus  into  new  concepts  where  problems 
appeared  to  be  different.  A  set  of  problems  was  developed  using  material  from  other 
disciplines.  The  intention  was  not  to  teach  the  scientific  or  engineering  concepts,  but  to 
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concentrate  on  the  mathematics  being  used  to  develop  the  concept.  The  problems  were 
collected  by  interviewing  faculty  members  in  person  and  by  email. 

Methods  of  Assessment:  There  are  two  levels  of  assessment:  assessment  of  student  work 
and  evaluation  of  the  new  courses.  Exams  reflected  the  new  curriculum  and  emphasized 
probleni  solving  rather  than  mechanics.  Tn  the  semester  that  the  switch  was  made  from 
the  old  courses  to  the  new,  a  common  portion  of  the  final  examination  in  the  old  Calculus 
III  and  new.  Calculus  II  revealed  that  the  students  were  able  to  learn  the  multivariate 
material  earlier  in  their  careers.  That  common  final  exam  was  written  by  an  instructor 
who  was  not  teaching  either  of  the  two  courses  and  was  developed  from  a  template  that 
divided  problems  into  conceptual,  mechanical,  and  applications.  A  Calculus  I  problem 
was  included  on  the  exam.  Most  students  did  not  recognize  its  simplicity  and  wanted  to 
make  it  more  complicated  than  it  was.  The  Physics  department  administered  a 
Mathematics  diagnostic  examination  that -reiterated  that  terminology  is  often  the  problem. 
Students  completed  evaluative  questionnaires  each  semester.  Each  faculty  member-in  the 
Mathematical  and  Computer  Sciences  Dep^ment  completed  a  survey  this  year  that  is 
presently  being  processed. 

Feedback  Mechanisms:  A  workshop  was  held  for  teachers  from  the  local  high  schools 
and  community  college  to  share  results  about  the  new  curriculum.  A  web  page  has  been 
established  to  help  facilitate  this  process.  The  results  of  the  faculty  survey  will  be  shared 
at  a  faculty  meeting  in  the  fall  and  necessary  changes  will  be  made.  The  student 
questionnaires  reveal  that  the  students  like  the  problem  solving  sessions  and  feel  that 
they  have  deepened  linderstanding.  Both  faculty  and  student  smveys  will  be  used  to 
provide  continuous  improvement.  Surveying  other  departments  for  input  is  a  process 
used  in  developing  the  new  computer  science  course,  differential  equations,  probability 
and  statistics,  and  numerical  methods.  We  plan  to  follow-up  by  asking  those  departments 
if  they  notice  changes  in  students’  performance. 

Goals/ABET  Criterion  Addressed;  The  session  addresses  ABET  Criterion  3  a) 
“Engineering  programs  must  demonstrate  that  their  graduates  have  an  ability  to  apply 
knowledge  of  mathematics,  science,  and  engineering.” 

Presenter: 

Name:  Barbara  Blake  Bath 
Address:  Colorado  School  of  Mines 

Department  of  Mathematical  and  Computer  Sciences 
Golden,  CO  80401 
Phone:  303-273-3872 
Fax:  303-273-3875 
Email:  bbath@mines.edu 
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Profile  of  the  Colorado  School  of  Mines  Graduate 

•  All  CSM  graduates  must  have  depth  in  an  area  of  specialization,  enhanced  by 
hands-on  experiential  learning,  and  breadth  in  allied  fields.  They  must  have 
the  knowledge  and  skills  to  be  able  to  recognize,  define  and  solve  problems 
by  applying  sound  scientific  and  engineering  principles.  These  attributes 
uniquely  distinguish  our  graduates  to  better  function  in  increasingly 
competitive  and  divejfse  technical  professional  environments. 

/ 

•  Graduates  must  have  the  skills  to  communicate  information,  concepts  and 
ideas  effectively  orally,  in  writing,  and  graphically.  They  must  be  skilled  in 
the  retrieval,  interpretation  and  development  of  technical  information  by 
various  means,  including  the  use  of  computer-aided  techniques. 

•  Graduates  should  have  the  flexibility  to  adjust  to  the  ever-changing 
professional  environment  and  appreciate  diverse  approaches  to  understanding 
and  solving  society’s  problems.  They  should  have  the  creativity, 
resourcefulness,  receptivity  and  breadth  of  interests  to  think  critically  about  a 
wide  range  of  cross-disciplinary  issues.  They  should  be  prepared  to  assume 
leadership  roles  and  possess  the  skills  and  attitudes  which  promote  teamwork 
and  cooperation  and  to  continue  their  own  growth  through  life-long  learning. 

•  Graduates  should  be  capable  of  working  effectively  in  an  international 
environment,  and  be  able  to  succeed  in  an  increasingly  interdependent  world 
where  borders  between  cultures  and  economies  are  becoming  less  distinct. 
They  should  appreciate  the  traditions  and  languages  of  other  cultures,  and 
value  diversity  in  their  own  society. 

•  Graduates  should  exhibit  ethical  behavior  and  integrity.  They  should  also 
demonstrate  perseverance  and  have  pride  in  accomplishment.  They  should 
assume  a  responsibility  to  enhance  their  professions  through  service  and 
leadership  and  should  be  responsible  citizens  who  serve  society,  particularly 
through  stewardship  of  the  environment. 
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Table  1:  Summary  of  Goals,  Objectives,  and  Evaluation  Methods 
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Calculus  Survey 


Because  of  comments  about  the  mathematical  abilities  (or  lack  of)  of  students  exiting  the 
calculus  program  at  CSM,  we  are  trying  to  develop  a  set  of  calculus  problems  which^are 
directly  related  to  concepts  seen  in  courses  in  all  other  disciplines  on  campus.  We  have 
selected  faculty  members  from  whom  we  feel  we  will  receive  helpful  input.  We  would 
appreciate  your  responses  to  these  questions. 


1)  Would  you  provide  a  specific  list  of  mathematical  topics  or  skills  which  pose 
difficulty  for  your  students? 

■J 

2)  Do  you  feel  that  your  student?  have  forgotten  the  math,  never  learned  the  math  or 
are  confused  about  the  change  in  vocabulary  or  symbols? 

3)  Do  you  find  yourself  re-teaching  math? 

a)  If  so,  how  much  time  do  you  spend  on  this? 

b)  If  not,  do  you  ask  students  to  review  the  concepts  on  their  own? 

4)  Would  you  be  willing  to  contribute  some  problems  from  subjects  you  are  teaching 
whose  solutions  involve  the  use  of  calculus?  We  would  appreciate  having  this 
input  as  soon  as  possible. 


We  appreciate  yoiir  time  in  helping  us  with  this  project.  We  feel  the  students  who 
worked  on  these  problems  last  semester  really  benefited  from  that  experience.  Thanks. 
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7.12  Flow  of  a  Gas  Mixture  through  a  Tank 


Calculus  Topic: 
Department:. 
Subject  Area: 
Time  Needed: 
Reference; 


Differential  Equations 
Chemical  Engineering 
Process  Simulation  and  Analysis 
40  minutes 


(30] 


When  a  gas  is  placed  in  a  tank,  it  expands  to  fill  the  whole  volume  of  the  tank.  Because 
of  this  pressure  must  be  considered  as  the  measurable  representation  in  this  problem.  The 
ideal  gas  law 

PV  =  jiRT  (7.28) 

can  be  used  to  relate  pressure  to  the  other  conditions  in  the  tank.  In  (7.28),  P  is  the  pressure 
in  the  tank,  V  is  the  total  volume  of  the  tank,  n  is  the  total  numbe’r  of  mols  of  gas,  R  is  a 
gas  constant  and  T  is  the  absolute  temperature. 

Consider  the  tank  system  in  Figure  7.23  which  has  a  two-component  ideal  gas  mixture 
at  a  constant  temperature.  The  gas  mixture  is  being  added  to  the  tank  at  a  flow  rate  (mols 


Figure  7.23:  Ideal  gas  mixture  at  a  constant  volume  and  temperature. 


per  amount  of  time)  of  F,  with  a,-  being  the  percent  of  the  composition  of  component  A. 
The  flow  rate  of  the  gas  out  of  the  tank  is  Fg  with  a  being  the  percent  of  the  composition  of 
A  leaving  the  tank  since  a  is  the  percent  of  the  composition  of  A  in  the  tank.  In  the  tank 
at  any  time,  there  are  N  mols  of  gas  mixture  of  which  mols  are  of  composition  of  A;  at 
pressure  P  and  a  is  the  percent  of  the  compositon  of  A  in  the  tank.  From  (7.28)  we  can  find 
that 

PV  aPV 

A-  =  -^  so  that  for  A  =  oA  = 

Considering  an  overall  mole  balance  for  the  system. 


(7.29) 
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The  mole  balance  for  component  A  is 


=  a,Ff  —  aFo 


(7.30) 


Troublesome  Notation:  The  differential  notation  is  also  confusing  when  substitution 

d—  pv  jfPv 

takes  place.  For  example,  —  =  — ^  is  a  simple  substitution  of - for  N.  — ^  simplifies 

Yfjip  di  dt  RT  dt 

to  -p  -  ■  since  V,  R  and  T  are  constants. 

Hi  (}t  *. 

* 

1.  Let  F;  =  0  so  that  fhe  tank  is  being  emptied.  Let  the  gas  leave  the  tank  at  a  rate 
proportional  to  the  difference  between  the  tank  pressure  and  the  pressure  of  the  atmo¬ 
sphere,  Patm^  ^  constant.  Then  the  overall  mole  balance  equation  is 


(~) 

\rt) 


where  the  ideal  gas  law  is  used  to  write  N  is  terms  of  P.  Considering  that  V,  T  and 
R  are  constant  and  that  when  t  =  0,  the  pressure  is  Po,  find  the  pressure  as  a  function 
of  t  at  any  time. 

2.  Now  consider  a  non  ideal  gas  which  behaves  according  to  the  non  ideal  gas  law 


PV  =  ZnRT  where  Z  =  1  + 


(7.31) 


where  B  represents  the  degree  of  departure  from  non-ideality  at  low  pressures  and  is 
constant  for  constant  temperature  processes.  As  in  problem  (1),  Let  P,-  =  0  so  that  the 
tank  is  being  emptied,  but  this  time  assume  that  the  tank  is  emptying  at  a  constant 
rate,  that  is,  P,  is  constant.  If  the  pressure  in  the  tank  is  P,  at  t  =  0,  set  up  the 
differential  equation  and  solve  for  P. 
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CURRICULAR  INNOVATIONS  OUTCOMES  ASSESSMENT 


ABSTRACT 


This  presentation  will  describe  how  the  Manufacturing  Engineering  Education  Partnership  (MEEP) 
esign^  ite  assessment  strategy  to  evaluate  the  outcomes  of  a  curricular  innovation  project  called  The 
Leaning  Jfactoij.  A  total  of  9  assessment  instruments  developed,  some  in  collaboration  with  industrial 
papers,  utilized  for  assessing  overall  and  specific  qualitative  aspects  of  the  program  as  well  as  student 
performance  (e.g.,  teamwork  skills  and  oral  presentation/written  skills)  are  described.  We  believe  that 

^  assessment  strategy  and  tools  used  comply  with  the  new 

ABET  Engineermg  Cnteria  2000'(criteria  2  and  3). 

SUMMARY  '/ 


The  _  Manufacturing  Engineering  Education 
Partnership  (MEEP),  a  coalition  of  institutions  who 
in  response  to  industry  needs,  has  developed  an 
innovative  manufacturing  engineering  curriculum 
option  and  physical  facilities  for  product  realization 
(See  Figure  1),  This  program  offers  a  new  paradigm 
for  engineering  education,  providing  a  balance 
between  theory  and  practice  and  emphasizing  the 
development  of  basic  skills  in  the  student  The 
desired  skills  include  communication,  teamwork, 
business  concerns  and  project  management.  Detailed 
information  about  the  program  can  be  found  in  the 
website,  http://lfserver.tf.psu.edu/LF/col  home.html; 
publications  can  be  requested. 


Figure  1.  MEEP  Curriculum  Model ' 
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A  CD-ROM  with  curricular  materials  and 


ASSESSMENT  STRATEGY  AND  TOOLS 


Developing  MEEP’s  assessment  strategy  proceeded  rather  easy  because  the  project’s  goals  and  objectives 
had  been  clearly  defined  in  the  project’s  Strategic  Plan.  An  assessment  team  was  formed  and  the  strateev 
discussed  and  shared  with  all  the  constituents  (faculty,  students,  and  industrial  partners).  Because  the 
pnting  agency  (NSF)  already  had  specified  the  quantitative  data  to  be  gathered,  the  assessment  strategy 
focus^  on  the  quaht^ive  aspects  of  the  program.  Once  the  project’s  goals  were  outlined,  four  matrices 
were  dewloped  (one  for  each  of  the  project’s  tasks)  which  contained  general  and  specific  questions  we 
thou^t  die  project’s  constituents  wanted  to  be  answered.  Table  1  presents  a  sample  from  one  of  the 
matacw  created.  The»  matrices  helped  the  assessment  team  develop  the  data  collection  approach  and 
design  the  assessment  instruments/  tools  for  the  different  audiences.  Assessment  tools  are  also  included. 

ASSESSMENT  INSTRUMENTS/TOOLS 

^  ^s  section,  several  of  the  assessment  instruments/tools  utilized  are  presented  in  three  categories: 
Projeci/Program  ^sessment  Toohj  Sindeni  Performance  Assessment  Tools,  and,  Course  and 
Curricular  Materials  Assessment  Tools.  Some  of  the  instruments  were  used  coalition-wide  and  others 
were  used  at  one  or  more  of  the  partnership  universities.  Some  of  the  tools  (e.g.,  surveys,  focus  group 
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questions)  were  developed  witii  the  help  of  our  industrial  partners.  Assessment  results  have  been 
published  elsewhere.' 

Proiect/Proeram  Assessment  Tools 

•  Stoy^s:  Four  surveys  were  developed  from  the  assessment  matrices,  focused  on  differeqt 
audiences:  students,  faculty,  industry  and  other  institutions.  Issues  and  items  in  the  surveys 
reflected  some  of  the  W;ays  in  which  MEEP  could  be  described.  Questions  ranged  from 
individual  perceptions  of  the  quality  of  specific  courses  and  activities,  to  faculty  evaluations, 
relationship  with  industry,  to  more  general  questions  surveying  the  overall  impact.  (See 
industry  survey). 

•  Industry/Faculty  Focus  Gro«/?:  ^Faculty  and  industrial  partners  frCm  the  three  institutions 
discussed  their  experiences  and  their  perceptions  as  to  what  made  the  partnership  a  success.  A 
discussion  group''Was  created  on-line,  and  opinions  shared  and  gathered  for  a  period  of  two 
months. 

•  External  Assessors:  A  group  of  experts  -  who  either  had  experience  in  .m^ufacturing 
engineering,  or  were  familiar  with  our  work  or  with  similar  partnerships/  learning  goals  - 
evaluated  the  project’s  deliverables.  They  participated  in  partnership  meetings,  talked  to 
industiy  partners,  students  and  faculty,  visited  facilities,  completed  the  survey,  or  browsed 
course  materials  in  national  conferences  and  meetings. 

Student  Performance  Assessment  Tools 

•  Teamwork  skills  assessment  instrument:  In  order  to  assess  the  students*  performance  in 
working  in  teams,  an  assessment  instrument  or  form  was  developed.  The  form  asked  students 
to  explain  their  decision-making  process  during  a  specific  task  they  had  to  achieve  (for 
example,  design  phase)  and  their  strategies  to  solve  conflicts  in  design  teams.  Besides 
assessing  student  performance  for  grading  purposes,  this  tool  helped  faculty  to  detect  if 
students  needed  more  training  on  how  to  work  in  teams.  Answers  provided  by  the  students 
were  discussed  in  class. 

•  Peers  Evaluation  Form:  At  the  end  of  the  semester,  students  evaluate  peers  in  their  teams. 
Tliey  assess  each  team  member  in  terms  of  the  effort  (0-3)  and  the  grade,  they  assess  the  work 
(in  percent). 

®  Oral/written  communication  assessment  instruments/tools:  Two  assessment  tools  were  used 
to  evaluate  the  students’  oral  and  written  communications  skills.  These  forms  were  used  by 
faculty  as  well  as  peers  in  evaluating  student  oral  presentations  and  written  reports.  Feedback 
from  peers  was  provided  to  the  student  teams  at  the  conclusion  of  the  presentation  . 

Course  and  Curricular  Materials  Assessment  Tools 

•  Course  Evaluation  and  Assessment  of  Skills  and  Knowledge  Instrument:  In  order  to  evaluate 
the  masteiy  and  level  of  knowledge  and  skills  developed  by  the  students  in  MEEP  courses 
and  to  establish  the  effectiveness  of  lectures  and  experiences,  as  'VYell  as  course  logistics,  an 
assessment  instrument  was  designed.  The  faculty  member,  customizing  it  to  the  individual 
course  adapts  this  generic  template. 

»  Lecturer  Evaluation  Form:  Some  of  the  MEEP  courses  offered  at  UPRM  are  team  taught.  A 
lecturer  evaluation  instrument  was  designed  to  determine  each  individual  lecture’s 
effectiveness. 

•  CD-ROM  Curricular  Materials  Assessment  Tool:  One  of  the  products  of  the  program  is  a 
CD-ROM  with  all  the  curricular/course  materials  developed.  An  assessment  form  was 
included  in  the  CD-ROM  to  evaluate  the  contents  as  well  as  the  quality  of  the  materials  in  the 
CD-ROM. 


'  Lucny  Morcll  dc  Ramirez,  Josi  L.  Zayas,  John  Lamancusa,  Jens  Jorgensen,  The  Manufacturing  Engineering  Education  Partnership: 
Program  Outcomes  Assessment  Results,  Frontiers  in  Education  Conference  Proceedings,  Pittsburgh,  November,  1997. 
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tiniculum  Dtvtlopment  McUrtc  (sample) 


Question  1 :  Was  a  new  inlcrdisaplmary,  practicc«based  curriculum  which  emphasizes  the 
Interdependency  of  raanufecturing  and  design,  in  a  business  environment 
developed? 

iubquestlons  . 

Data 

Collection 

Approach 

Re^ndent 

(students, 

faculty, 

industry) 

Schedule*  P 

ia.  Did  the  prt^rani  allow  students  to  practice  their  engineering  science  fundamentals  in  the 
solution  of  real  problems? 

/ 

/ 

Questionn¬ 
aire  (Q)  or 
Focus  Group 
(FG) 

Samples 

s.  F.  I 

lb.  Arc  professional  communication  and  team  skills  taught  and  kamed? 

•/ 

QorFG 

Samples 

Interviews 

S.F.l 

. 

Ic,  Arc  case  studies,  active  learning  techniques,  and  computer  technologies  extensively  used  in  the 
classroom? 

QorFG 

Samples 

mfn 

[cTthd  program  provide  previously  unavailable  opportunities  for  hands  on  engineering 
experience  In  the  Learning  Factory? 

Hi 

Ic.  Did  the  partner  schooU  exchange  mformahon  and  IwTfiomcacn^hwYcxpcnScw? 

S.F,I 

If  Did  students  take  courses  with  students  from  disciplines  other  than  engineering? 

QorFG 

S 

Ig.  Did  faculty  develop  or  modify  courses  to  accommodate  multiple  engineering  disciplines? 

QorFG 

F 

Ouestk^n  i:  Was  a  new  paradigm  tor  coalition-wide  courses  development,  sharing  and  export  to 
the  aoukmic  community  at-large  developed? 

iSubquesdoDS 

Data 

Collection 

Approach 

Respondent 

s 

Schedule 

HIHI 

S,F.I 

1 2b.  Were  jointly  developed  curriculum  msterials  easily  transported  among  the  MEEP  partners,  and 
1  c?qx>rted  to  the  academic  community  at  large? 

■■ 

S.F 

QorFG 

Simples 

mm 

mmmm 
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INDUSTRY  SURVEY 


Th^  Learning  Factory  is  a  new  practice  based  curriculum  and  physical  facilities  for  product  realization  that  has  been  developed 
at  three  institutions:  Penn  State  University,  the  University  of  Washington,  the  University  of  Puerto  Rico  at  MayagOei  in 
collaboration  with  Sandia  National  Labs.  Its  goal  is  to  provide  an  improved  educational  experience  that  emphasizes  the 
interdepe^ency  of  manufacturing  and  design  in  a  business  environment.  The  key  element  in  this  approach  is  active  learning  - 
the  combination  of  curriculum  revitalization  with  coordinated  opportunities  for  application  and  hands  on  experience. 

This  questionnaire  has  been  designed  to  assess  the  perforrhance  and  products  of  this  program.  Please  ar^wer  it  to  The  best  of 
your  knowledge. 

_ -  --  ■ _  Company: _ _ 

Partner  University: 

IJUPR-M  []PSU  []UW  n  Other  _ 

Your  Involvement  with  the  program: 

[  ]  Member  of  Industrial  Partner  Board  [  ]  Expert  in  the  classroom  [  ]  Involved  with  students  projects 
[10thcr_ _ 

Instructions:  ~  — - - 

The  following  items  reflect  some  of  the  ways  in  which  the  Manufacturing  Engineering  Partnership  (MEEP)  can  be  described 
Please  fill  in  the  numbered  circle,  which  indicates  THE  DEGREE  TO  WHICH  YOU  AGREE  that  each  item  is  descriptiye  of  the 
experiences  you  were  exposed  to  and  provided  by  the  program.  If  you  have  no  information  or  feel  an  item  docs  not  apply  please 
fill  in  the  N/A  (Not  Applicable)  circle. 


Strongly  Agree  4  -  Agree  3  >  Neutral  2 -Disagree  1  >  Strongly  Disagree  N/A 


The  program  allowed  students  to  practice  engineering  science  fundamentals  in  the 
solution  of  real  problems. 

5 

4 

3 

2 

1 

—  j 

Professional  communications  skills  were  enhanced. 

T" 

4 

3 

2 

1 

N/A  1 

Teamwork  skills  were  enhanced. 

5 

4 

3 

2 

1 

N/A 

fhe  partner  schools  learned  from  each  other's  experience. 

5 

4 

3 

2 

1 

N/A 

Resources  and  ideas  were  shared,  avoiding  redundant  efforts. 

5 

4 

'3 

2 

1 

N/A  • 

Real  life  problems  were  provided. 

5“ 

4 

3 

2 

1 

N/A  11 

New  technologies  for  communication  were  utilized  on  curriculum  content 

5 

4 

3 

2 

1 

N/A  S 

Ibc  local  Industrial  Advisory  Board  (lAB)  provided  quality  strategic  and 
operation  guidance  to  the  local  institution. 

5 

4 

3 

2  . 

,1 

.  N/A 

Ilic  local  lAB  supported  MEEP's  activities  providing  fmancial  and/or  non 
financial  resources. 

~ 

4 

3 

'2 

1 

N/A 

There  was  good  communication  between  industrial  sponsors  and  the  institution. 

~ 

4 

3 

2 

1 

N/A 

Each  institution  provided  the  lAB  the  right  infonnation  in  a  timely  fashion. 

5 

4 

3 

2 

1 

N/A 

The  MEEPs  Industrial  Advisory  Board  (lAB)  evaluated  the  overall  progress  of 
the  program. 

5 

4 

3 

2 

1 

N/A 

llic  partnership  reported  progress  and  activities  related  to  participation  in 
curriculum  development. 

4 

3 

2 

1 

N/A 

Ihc  MEEPs  lAB  provided  support  in  actions/activitics  that  arc  relevant  to  the 

1  program. 

~ 

4 

3 

2 

1 

N/A  . 

r  Fhe  partnership  reported  progress  and  activities  related  to  participation  in  the 
classroom  teaching. 

5 

4 

3 

.2 

1 

Students  completing  the  MEEP  program  arc  more  useful  to  our  industry. 

5 

4 

3 

2 

1 

N/A 

My  industry  and  company  is  more  likely  to  hire  a  MEEP  trained  student  than  a 
traditionally  trained  student 

5 

4 

3 

2 

1 

N/A 

Would  you  encourage  other  companies  to  participate  in  the  program  and  coalition?  Why? 


What  can  be  improved  with  MEEP? 


Comments: 
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University  of  Puerto  Rico  ~  ~  - 

MayagQez  Campus 

ADMI 3100  -  TECHNOLOGY  BASED  ENTREPRENEURSHE? 

TEAMWORK  EXPERIENCES  ASSEjiSMEOT  FORM 

Please  answer  the  following  questions  regarding  your  work  as  a  team  for  the  completion  of  the  required  task. 

TASK(S):  PRODUCT  DESIGN.  DECISION-MAKTNO 

1.  In  chronological  order,  list  what  your  team  did  during  the  design  phase.  Explain  how  tasks  were  distributed,  how  decisions 
were  made.  / 


2.  What  facilitated  the  decision-making  process? 


3.  What  was  your  contribution  to  the  team  when  decisions  had  to  be  taken? 


4.  What  do  you  think  you  would  like  to  do  differently  the  next  time  when  working  in  a  team? 


name _  TEAM 
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MANUFACTURING  ENGINEERING  EDUCATION  PARTNERSHIP 
University  of  Puerto  Rico -Mayagtiez 

COURSE  EVALUATION  and  ASSESSMENT  OF  SKILLS  and  KNOWLEDGE 

Course: _ Instructor: _ _ 

The  purpose  of  tiiis  assessment  is: 

•  to  determine  your  perception  of  mastery/levcl  of  knowledge  and  skills  developed  by  the  students  in  this 
course,  and 

•  to  establish  the  effectiveness  of  lectures  and  experiences,  as  well  as  of  die  logistics  used. 

The  results  of  this  assessment  will  help  die  instructor  in  charge  of  die  course  to  better  plan  and  adjust  the  course’s  agenda 
in  the  future.  ' 


PART  I:  GENERAL  OBJECTIVES  AND  SKILLS 
Dictions: 

Using  the  scale  below,  please  evaluate  (*)  yoiir  perception  of  the  masteiy  of  skills  and  experience  the  students 
developed  in  this  course  in  the  areas  specifi^. 


N:  no  skills/no  experience 
R:  rudimentary  skillsA^ery  little  experience 
F:  functionally  adequate  skills/some  experience 
A:  advance  skill/extensive  experience 


area 

* 

skill  1 

skill  2 

objective  1 

objective  2 

PART  H:  CONTENT,  LECTURES  AND  EXPERIENCES 
Directions: 

In  this  part,  please  indicate  (♦)  your  perception  of  the  lectures  and  activities*  effectiveness,  using  the  following  scale: 

0:  not  effective;  would  eliminate 

1:  moderately  effective;  significant  changes  (specify) 

2:  effective;  minor  changes  (specify) 

3:  very  effective;  would  not  change 
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PART  ni:  COURSE  LOGISTICS 
Directicms: 

Please  indicate  (*)  how  you  feel  regarding  the  various  aspects  designed  for  the  course,  using  the  following 
scale: 

0:  inadequate;  disliked,  needs  re-engineering! 

1:  somewhat  adequate;  needs  enhancement 
2:  adequate;  minor  changes 
3:  adequate;  no  change 


area 

PH 

comments  I 

Number  of  meetings  / 

Hi^ 

Kinds  of  assessment  techniques 

Requirements 

■1 

Humber  of  lectures 

■ 

Number  of  plant  trips 

Tc^ics  covered 

' 

Course  coordination 

Other. 

1 

Would  you  recommend  this  course  to  odicr  students?  Explain. 


Do  you  diink  your  expectations  were  met?  YES/NO.  Explain. 


Suggestions: 


Your  overall  rating  of  the  course: _ /lO, 
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The  Manufacturing  Engineering  Education  Partnership  (MEEP) 
CD-ROM  Assessment  Form 

Please  review  this  CD-ROM  and,  to  the  best  of  your  knowledge,  answer  the 
questions  that  follow  regarding  the  contents  and  quality  of  the  curricular 
materials  included.  We  would  also  like  to  know  how  useful  these  materials 
could  be  to  you  or  to  any  institution  willing  to  adopt  or  adapt  them.  Your 
feedback  will  help  the  Partnership  in  its  effort  to  fine  tune  the  curricular 
products  developed. 


Name 

Position 

Institutio 

n 

• 

Address 

Phone:  Fax:  email: 

The  MEEP  CD-ROM  contains  the  following  items: 
Background  Information 

•  Information  about  MEEP 

•  Video 

•  MEEP  Publications 
Course  Materials 

•  Product  Dissection  Course 

•  Technology-based  Entrepreneurship  Course 

•  Concurrent  Engineering  Modules 

•  Process  Quality  Engineering  Course 

•  Rapid  Prototyping  Technology  Module 


I.  Regarding  Background  Information: 

Did  you  understand  the  program,  as  described  in  the  Information  about  MEEP  section? 


Was  the  video  about  the  program  useful  in  understanding  the  goals  and  objectives  of  the  Partnership? 


Did  the  publications  about  MEEP  provide  more  details  about  the  different  aspects  of  the  program  (e.g. 
goals,  approach,  products,  assessment)? 

Cuniojlar  Innovations  Outcomes  Assessment 
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Regarding  the  Course  Materials: 


How  would  you  rate  the  content  and  quality  of  the  course  materials? 
Use  the  following  rating:  1  (poor);  5  (excellent) 


Content 

Quality 

Comments 

Product  Dissection  Course 

Entrepreneurship  Course 

Concurrent  Engineering  Modules 

Process  Quality  Engineering  ' 
Course 

Rapid  Prototyping  Technology' 
Module  / 

III.  Regarding  the  use  of  the  contents  of  the  CD-ROM 


Will  you  use  the  curricular  materials  included?  If  the  answer  is  yes,  how  would  you  use  them? 


I 
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Professor  of  Chemical  Engineering  and  Director  of  the  Curriculum  Innovation  Center  of  the  Puerto  Rico 
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DEVELOPING  A  SCHOOL-WIDE  ASSESSMENT  PLAN: 
LESSONS  LEARNED  AND  QUESTIONS  RAISED 

Goalf/ABET  Criterion  3  Addressed 

This  presentation  addresses  ABET  EC2000's  Criterion  2.  Program  Educational 
Objectives,  and  Criterion  3,  Program  Outcomes  and  Assessment.  We  describe  the  process  that 
has  led  our  s^ool  from  a  state  of  outcomes  assessment  (OA)  resistance  to  outcomes  assessment 

^ceptance.  This  session  is  intended  for  anyone  who  is  involved  in  moving  assessment  forward  in 
his  or  her  organization. 

Presentation  Format  *  / 

This  presentation  will  use  a  panel  discussion  format  with  open-ended  invitation  for 
confere^  attends  to  contribute  to  the  discussion  throughout  the  presentation.  Panelists  from 
me  ^due  School  of  Engineemg  and  Technology  at  lUPUI  will  present  the  responses  of  five 
faculty  constituencies  to  questions  that  we  have  had  to  answer  as  we  made  the  transition  from 
itmtmx  to  Mceptaoce.  The  five  faculty  constituencies  include  the  department  chairs,  the  dean's 

otBce,  the  junior  faculty,  the  resistant  faculty,  and  the  chair  of  the  school-wide  assessment 
committee. 

Session  Summair 

Our  story  will  unfold  in  a  Frequently  Asked  Questions  (FAQ)  format,  with  panelists 
presenting  and  honest  responses  to  series  of  questions  that  have  been  carefully  selected  and 
arranged  to  pamt  a  complete  picture  of  our  transformation  from  resistance  to  acceptance.  This 
presentation  will  bring  into  the  open  what  is  often  heard  only  in  private  conversations  We  will 
describe  (1)  the  resistance  and  objections  that  we  have  had  to  overcome,  (2)  the  questions  we 
have  had  to  aiiswer,  (3)  the  ciment  level  of  cooperation  and  faculty  buy-in,  (4)  the  lessons  we 
have  learned,  (5)  the  organization  of  the  faculty  over  sight  committee  that  facilitated  the 
transformation,  and  (6)  the  support  we  have  received  from  the  dean’s  office  and  the  campus 
admimstratioa  In  addition,  we  will  describe  (7)  our  accomplishments  to  date,  and  (8)  we  wfll 
present  a  list  recommendations  for  deans  and  chairs  of  faculty  leaders.  Finally,  (9)  we  will 
conclude  wiffi  a  disclosure  of  questions  that  we  have  yet  to  answer,  bringing  the  audience  into  the 
program  to  brainstorm  possible  resolutions  to  the  questions. 

The  tables  of  infonnation  presented  in  this  session  were  condensed  from  a  more 
comprehensive  set  due  to  the  time  allotted  and  the  number  of  pages  allowed.  We  have  produced 
a  brochure  that  contains  the  complete  tables  and  other  information  that  may  be  useful  to  you.  A 
URL  for  a  web  site  containing  this  brochure  will  be  announced  at  the  Symposium.  If  you  did  not 
receive  ^  handout  or  the  URL,  you  may  contact  Chartes  Yokomoto  at  the  address  listed  below 
A  rotated  paper  can  also  be  found  in  the  proceedings  of  the  1998  Frontiers  in  Education 
Conference  [1]. 
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Key  Words 

Outcomes  assessment,  culture,  resistance,  buy-in,  assessment  process,  school- wide  assessment 
committee,  support 
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DEVELOPING  A  SCHOOL-WIDE  ASSESSMENT  PLAN: 
LESSONS  LEARNED  AND  QUESTIONS  RAISED 
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TABLE  10.  WHAT  HAS  OUR  DEAN  DONE  TO  SUPPORT  OUTCOMES  ASSESSMENT? 
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TABLE  14.  WHAT  DIFFICULTIES  DID  WE  HAVE  WITH  PUBLISHED  PROCESSES 


TABLE  19.  UNEXPECTED  FALLOUT  OF  PLANNING  THIS  PRESENTATION 
The  discussions  that  we  had  in  puttinj-,  the  program  together  helped  further  our  understanding  of  outcomes  assessment. - 
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Electronic  Portfolios  >  the  Technical  Side 


Goals/ABET  Criterion  (Criteria)  Addressed 

Wifli  the  emphasis  on  outcome-based  assessment  (sec  ABET  Engineering  Criteria  2000: 

Criterion  3— Program  Outcomes  and  Assessment),  portfolio  assessment  provides  a  framework 
for  documenting  and  evaluating  student  outcomes.  This  session  vdll  provide  participants  an 
overview  of  thie  development  process  of  an  electronic  portfolio  system  prototyped  at  Rose- 
Hulman  Institute  of  Technology.  The  foci  of  this  presentation  are  to  discuss  various  issues 
addressed  during  the  system  epgineering  process,  and  to  illustrate  how  this  system  vdll  satisfy 
the  specified  objective  (facilitating  the  documentation  and  evaluation  processes  of  student 
outcomes). 

/ 

Presentation  Format 

This  session  will  consist  of  a  presentation  using  PowerPoint  Slideshow  and  follow  by  a  question 
and  answer  session. 

Session  Summary 

Portfolio  assessment  is  chosen  at  Rose-Hulman  as  the  primary  method  to  collect  evidence  of 
student  outcomes,  and  an  electronic-version  of  the  portfolio  assessment  was  prototyped  during 
the  winter  quarter  of  AY  1997-98  with  30  students  representing  the  sophomore  class. 

The  first  step  in  building  an  electronic  portfolio  system  begins  with  the  requirement  analysis. 

We  ask  ourselves  these  questions  to  identify  which  direction  of  the  system  engineering  process 
would  go: 

A)  Who  is  the  audience? 

B)  What  is  the  context? 

C)  What  is  being  measured? 

D)  What  constraints  are  present? 

E)  What  resources  are  available? 

Recognizing  the  rapidly  evolving  technology,  a  conscious  decision  has  been  made  to  apply  open 
systems  principles  throughout  the  system  engineering  process.  In  an  attempt  to  avoid  major 
redesign  of  the  system  in  the  future,  modular  design/rapid  prototyping  approach  helps  to  ensure 
the  system  can  be  quickly  built,  and  yet  supported  with  changing  products  and  technologies. 
Interface  management  is  also  critical  in  supporting  technology  insertion.  The  "web-database" 
interface  is  selected  to  help  achieving  our  goal  because  of  its  platform-ind^ndent  feature. 

Once  these  decisions  are  finalized,  the  system  design  and  functional  analysis  will  then  follow. 
Each  of  the  three  major  system  components  (Input,  Output  and  Process)  has  been  examined 
carefully  to  best  utilize  available  resources  with  given  constraints.  In  general,  the  input 
component  of  the  electronic  portfolio  system  covers  materials  submission,  modification  of 
existing  infonnation  in  portfolio  option,  on-line  rating  and  feedback  options  (to  be  implemented), 
and  end-user's  preference  selection  (to  be  implemented).  Searching,  extracting  and  viewing 
items  from  portfolios,  and  generating  ad  hoc  and  routine  reports  (to  be  implemented)  are  within 
tiie  output  component.  The  process  component  is  the  "brain"  of  the  system  which  controls 


F)  What  should  be  included  in  a  portfolio? 

G)  How  should  a  portfolio  be  organized? 

H)  In  what  capacities  will  portfolios  be 
utilized? 
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security  handling  of  the  portfolios,  keeps  track  of  versions  and  defines  relationship  between  the 
identified  objectives  and  submitted  materials,  logs  transaction  activities,  joins  student 
background  information  from  campus  legacy  system  (to  be  implemented),  nms  queries  and  posts 
search  results. 

Addressing  the  needs  identified  during  tiie  requirement  analysis  stage,  the  result  of  the  system 
engineering  process  produces  a  student-centered,  organized,  easy-to-leam,  easy-to-use  electronic 
portfolio  system. 

jf. 

Key  Words 

Electroriic  Portfolios,  Interface  Management,  Open  Systems  Principles,  Rapid-prototyping, 
Requirement  Analysis,  Student-Centered,  System  Components,  System  Engineering  Process. 
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Learner-Centered  Web-Based  Assessment  Tool 
for  Large  and  Small  Classes 

Goals/ABET  Criteria  Addressed 

The  classroom  assessment  technique,  which  is  a  part  of  every  class  period,  can  be  a  key 
component  of  a  broader  assessment  plan  satisfying  Criteria  1-3  of  the  EC  2000. 

Presentation  Format 

Interactive,  video  clips,  demonstration  via  the  Internet  (if  possible). 

Session  Summary  , , 

The  primary  purpose  ofolassroom  assessment  is  to  determine  how  well  students  are 
learning  on  a  continuous  basis  and  to  take  necessary  corrective  action  as  soon  as  possible 
to  improve  their  learning.  This  presentation  will  describe  a  computerized  web-based 
assessment  technique  that  can  be  used  effectively  and  easily  in  large  and  small  classes. 
The  technique  allows  instructors  and  students  to  monitor  the  learning  process  on  a  daily 
basis  and  to  quickly  point  out  the  need  for  corrective  action,  if  necessary.  Students  can 
view  their  performance  arid  also  the  average  class  performance  on  the  web.  The  graphical 
displays  allow  them  to  see  the  trend  of  their  performance  over  several  days.  Several 
reports,  automatically  generated  for  instructors,  allow  them  to  determine  promptly  how 
well  certain  concepts  or  topics  are  learned  by  students.  Use  of  special  codes  and  optical 
scanning  sheets  minimize  paper  shuffling  in  large  classes,  and  the  computer  program 
automatically  grades  and  updates  the  database.  The  program  is  fairly  general  and  has 
friendly  interfaces;  thus  it  can  be  implemented  for  use  by  faculty  at  other  universities. 
According  to  student  response  sheets,  80  to  97  percent  of  the  students  find  different 
aspects  of  the  technique  to  be  effective  in  improving  learning. 

Key  Words 

Classroom  assessment,  web-based  assessment,  daily  homework,  reading  quizzes, 
attention  quizzes,  prompt  feedback,  large  classes 
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Oiifline  of  an  Interactive  Presentation 

1.  What  is  classroom  assessment? 

2.  How  is  it  related  to  ABET’s  EC  2000? 

3.  What  is  the  importance  of  prompt  feedback? 

4.  How  to  handle  daily  homework,  especially  in  large  classes? 

5.  What,  Why,  and  How  to  conduct  Reading  Quizzes? 

6.  What,  Why,  and  How  to  conduct  Attention  Quizzes? 

7.  How  DHQM  can  help  in  small  or  large  classes? 

8.  Would  you  like  to  see  a  short  video  on  DHQM  in  action? 

9.  What  are  the  results  of  the  DHQM? 

10.  Is  DHQM  available  to  other  schools? 

11.  Would  you  like  to  see  a  demonstration  of  the  DHQM? 
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DHQM 

Daily  Homework/Quiz  Manager 


Welcome  to  the  Daily  Hoiiework/Quiz  Management  home  page.  Please  follow  one  of  the 
following  links: 

•  Grade  Information  ( <-  Click  here  if  you  are  a  student!) 

•  Instructor  Pages 

o  Register  as  an  instructor  for  DHQM  here! 

•  To  preview  the  Instructor  Pages,  click  on  the  "Instructor  Pages"  link  and  enter  in  'test'  for  both 
the  Username  and  Password. 


•  About  the  DHQM  Package 


Figure  1  The  first  Screen  of  the  DHQM 

(You  can  test  the  package  by  going  to  http://valley.nodak.edu/dhqm/) 
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DHQM  -  Class  Management 


Process  new  OMR  files  Help 


Daily  Suinmai-y  (for  a  particular  davl  I  Help 


Multi-Dav  Summary/  i  Help 


•  Add  a  student  to  your  class  Help 
e  Delete  a  student  from  your  class  !  Help  | 


Change  a  student’s  grade 


Help 


Add  a  new  class  Help 


Delete  a  class 


Help 


Delete  the  data  for  a  particular  day  (95%)  Help 


Download  Class  Database  (Exceii/Tab  Deiim.) 


Help 


Rack  to  the  Main  DHQM  page 

Ahnnt  the  DHQM  Package 

Figure  2  Features  available  on  the  Instructor’s  Page 
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DHQM  Daily  Summary 


Class:  me221 

Select  the  date  that  you  wish  to  view: 


August  29, 

1997  1 

September 

3. 

1997 

September 

5, 

1997 

September 

8, 

1997 

September 

12, 

1997 

September 

15, 

1997/  > 

September 

17, 

1997/ 

Continue 


Change  Class 


Summary  for  December  3, 1997 


Frequency 

Distribution 


Homework  question:  1 


Quiz  question:  2 


Mean: 


Standard  Deviation: 


Frequency 

Distribution 


1 

1(  0.279 

Score: 

Number  of  scores:  1 

0 

OSlESli 

1 

JSSBBSBtik 

11^^ 

Figure  3  A  Part  of  Instructor’s  Daily  Report  Showing  Statistics  for  Homework  and  Quiz 
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udent  Grades 


Overall  Grade 


Homework 


Your  Total  Grade;  428/570  (75%) 

Class  Total  Average:  554.2  /  570  97.23  % 


1 

Quiz 

32/61 

(52%) 

149.8/61 

81.66  % 

Daily  Grades 


Class  Date 


November  25, 
1996 

124  students 


December  2, 
1996 

119  students 


December  4, 
1996 

121  students 


Homework 


Your  Score:  ^  (100%) 


Quiz 


Class  Average: 


Your  Score:  ^  (100%) 


fnWrMjiJii  UL 


Class  Average: 


Your  Score;  1^^  ^  (100%) 


2/2  (100%) 


0 


1  /  2  (50%) 


1.615 


Back  to  the  Main  DHQM  page 
Back  to  the  Student  Login  page 


About  the  DHQM  Package 


Figure  4  Feedback  to  Students  about  their  Daily  and  Overall  Performance 
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PROCESS  FOR  CURRICULUM  ASSESSMENT 
IN  MECHANICAL  ENGINEERING 
AT  THE  UNIVERSITY  OF  WISCONSIN 


A 


Goals/ABET  Criterion  Addressed 

The  goal  of  the  session  is  to  aid  faculty  in  developing  a  plan  and  carrying  out  the  assessment  of  a 
program  in  support  of  both  ABET  visits  and  curriculum  development. 

Presentation  Format 

'  '  -  ^ 

The  format  tvill  be  presentatipn  in  segments  with  time  for  questions  and  answers. 

/  ■ 

Session  Summary 

The  experience  gained  in  involving  faculty  at  a  large  research-oriented  department  in  the 
assessment  of  student  education  will  provide  the  basis  of  the  discussion.  Faculty  interest  in  the 
assessment  process  is  crucial,  and  at  UW  the  faculty  had  already  made  an  implied  commitment 
through  their  involvement  in  several  learning  and  teaching  programs.  A  departmental 
Assessment  Committee  was  then  able  to  build  on  this  commitment.  At  retreats  and  faculty 
meetings  this  committee  lead  the  process  by  which  the  assessment  objectives  were  defined.  The 
LEAD  Center,  which  is  an  evaluation  center  on  campus,  took  the  Department  objectives, 
modified  them  to  yield  useful  results,  and  carried  put  the  actual  assessment.  The  experience  with 
the  process  of  generating  the  assessment  tools  with  the  entire  faculty  involved  and  the  results  of 
the  assessment  will  be  the  subject  of  the  workshop. 

Key  Words 

Assessment,  Evaluation,  Exit  Interviews,  Alumni  Surveys 
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Who  is  the  Mechanical  Engineering  Department? 

•  Department  of  35  faculty  members  in  three  areas  (Energy,  Design,  Manufactunng) 

•  Between  150  and  175  students  graduated  each  year  with  a  BS  degree 

•  220  graduate  students  at  the  MS  and  PhD  level,  most  supported  on  research  funds 

How  did  the  assessment  process  initiate? 

•  Assessment  of  curriculum  initialed  in  1995  based  on  the  Department  s  Strategic  Plan 

•  Process  began  with  small  groups  of  faculty  in  meetings  and  retreats 

•  Faculty  developed  a  set  of  educational  objectives  for  curriculum 

•  Initially,  a  set  of  Knowledge  and  Skill  goals  for  students  to  master  by  graduation 

•  Finally,  a  set  of  detailed  and  specific  questions  relating  to  mastery  of  material 

•  During  this  process  .  ,  ^ 

•  Faculty  knew  what  they  wanted  to  assess  but  were  unsure  of  how  to  do  it 

•  Faculty  developed  methods  of  assessment  that  would  be  difficult  to  implement 

•  Departmental  resources  became  a  limiting  factor  in  conducting  the  assessment 

•  Faculty  were  interested  in  assessment  but  attention  was  limited  to  about  2  hours 

•  Faculty  had  demonstrated  a  commitment  to  learning  and  teaching  programs: 

•  Faculty  Teaching  Improvement  Program 

•  Weighting  ofteaching  performance  in  merit  raises 

•  Development  of  a  teaching  evaluation  form 
How  was  the  assessment  actually  carried  out? 

•  The  faculty  as  a  whole  defined  the  objectives  and  issues  of  interest 

•  lead  Center  and  a  small  faculty  committee  converted  these  into  a  useable  tool 

•  Not  appropriate  for  faculty  to  conduct  student  interviews 

«  Interview  questions  could  not  address  techmcal  achievements 
o  Conventional  classroom  testing  was  appropriate  to  assess  knowledge  and  skills 
«  Necessary  external  funding  was  provided  by  the  UW  Assessment  Council 
.  Interviewed  25  graduating  seniors  in  class  of  1 997  (about  1 5  %  of  graduating  class) 

•  Questionnaire  on  quantitative  aspects 
e  Interview  on  qualitative  aspects 

•  Sent  surveys  to  the  1 57  students  of  the  class  of  1 994 

•  Covered  both  qualitative  and  quantitative  aspects  of  program 

•  Sixty  surveys  returned  (38  %  of  class) 

What  have  been  the  results  of  the  assessment  process? 

•  -Curriculum  improvements  in  response  to  student  consensus 
e  Advising  system 

o  Instrumentation  laboratory  course 
o  Curriculum  integration 
o  Time  to  graduate  and  number  of  credits 

•  Continued  improvement  of  assessment  process  for  class  of  1998 
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What  is  LEAD?  (Learning  through  Evaluation,  Adaptation,  &  Dissemination) 

•  Founded  in  1 994  at  the  UW-Madison  by  Dr.  Susan  Millar 

•  Research  center  of  professional  evaluators  trained  in  anthropology,  psychology,  education, 
mathematics,  etc. 

•  Provides  evaluation  research  support  for  educational  reform  efforts  at  both  the  xmdergraduate 
and  graduate  levels  and  in  support  of  university  outreach  activities 

•  Projects  include  course-level  evaluations,  program  (i.e.,  department  or  major)  assessments, 
and  institutional  evaluations 

•  Funding  for  LEAD  projects  is  through  grants  provided  by  clients  (these  grants  may  be 
secured  through  partnerships  with  LEAD  researchers) 

What’s  so  important  about  “third-party”  evaluation? 

•  Students  and  alumni  areinore  likely  to  share  information  with  external  evaluators  that  they 
might  be  hesitant  to  provide  directly  to  the  department 

•  Well-chosen  external  evaluators  come  to  the  project  with  enough  context  to  make  sense  of 
the  findings,  but  without  the  preconceptions  of  department  members 

•  Trained  evaluators  can  provide  expertise  and  experience  in  conducting  quantitative  and/or 
qualitative  research,  and  are  familiar  with  relevant  literature,  research,  and  resources 

What  did  LEAD  do  for  Mechanical  Engineering? 

•  Objectives  and  issues  of  interest  were  defined  by  ME  faculty 

•  LEAD  researchers  shaped  these  objectives  and  issues  into  effective  survey  and  interview 
questions 

•  LEAD  researchers  maintained  anonymity  of  students  and  alumni  by  serving  as  conduit  for 
information 

•  LEAD  researchers  analyzed  the  information  arid  foimd  central  themes: 

®  speech  training  available  only  in  on-campus  communications  class 

•  laboratory  too  rigorous  to  allaw  digestion  of  material 

•  advising  frequently  unused  because  of  department  “flow-chart" 

o  computer  skills  learned,  of  necessity 

«  "independent  learning"  was  an  implicit  theme  in  the  interviews  and  surveys 

What  can  a  department  do  vi'ithout  a  LEAD  Center? 

•  Build  a  rapport  with  your  students  (and  thus  your  alums).  If  they  know  you  want  and  will 
use  the  information  they  provide,  they  are  more  likely  to  help  out,  even  when  you  don’t 
explicitly  ask.  One  way  to  do  this  is  to  share  what  you  find  in  your  assessments  and  share 
what  you  are  doing  vrith  those  results. 

•  Test  your  survey  or  interview  questions  on  a  colleague  or  a  few  students  to  be  sure  the 
questions  are  clear  and  will  elicit  the  information  you  need. 

•  Allow  students  to  respond  anonymously  or  via  a  non-faculty  member. 

•  Read  between  the  lines:  look  for  patterns  in  what  students  are  saying  and  look  for 
connections  across  questions. 

•  Don’t  reinvent  the  wheel:  borrow  from  what  colleagues  have  done,  get  recommendations  on 
useful  and  relevant  literature  (see  the  bibliography  provided  above). 
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DEPARTMENT  of  MECHANICAL  ENGINEERING  ASSESSMENT 
PRE-INTERVIEW  SURVEY  -  GRADUATING  SENIORS 


1.  Name  - - - - - - - - - 

2.  How  many  semesters  has  it  taken  you  to  reach  graduation  (inch  co-op)? 

3.  How  many  credits  has  it  taken  you  to  reach  graduation?  - - - 


4.  Have  you-been  a  full-time  student? 

O  Yes,  throughout  school. ' 

O  No.  l’\e  always  been  part-time. 

O  Sometimes  full-time,  sometimes  part-time. 


5  Which  of  the  following  describe(s)  your  work  experience  during  the  past  year.  Check  all  that  pp  y 

o  7  \e  been  working  approximately _ hours  per  week  during  the  summer. 

O  I  have  completed  one  or  more  co-ops.  (How  many? - ) 

O  I  have  completed  one  or  more  internships.  (How  many?  _J 

.  .O  I  have  work  experience  related  to  engineering.  Which  f  el  (s).  - - - 

O  I  have  ndt  been  working  while  attending  school. 


6. 


What  are  your  post-graduation  plans?  (If  you  are  planning  on 
please  indicate  which  you  plan  to  do  first.) 

O  Employment  in  (what  field?) - - - 

O  Graduate  school  in  (what  field?)  - - - - - - — 

O  Undecided 


both  employment  and  graduate  school, 


7. 


Have  you  had  a  job  offer?  O  Yes  bio 

What  type  of  company(ies)  are  you  interviewing  with. 
O  Consulting  O  Sales  &  Service 
O  Design  O  Research  &  Development 


O  Not  sure 
Check  all  that  apply. 

O  Manufacturing 
O  Other 


8.  Where  would  you  like  to  work  after  graduation? 

O  Specific  city,  state,  region,  or  country  (specify  here) 
Q  Anywhere  I  can  get  the  best  Job. 

O  Undecided. 


9.  Have  you  taken  or  do  you  plan  to  take  the  FE  exam?  O  O  No 


O  Not  sure 
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DEPARTMENT  of  MECHANICAL  ENGINEERING  ASSESSMENT 
INTERVIEW  QUESTIONS  -  GRADUATING  SENIORS 

Note  to  Interviewers:  Begin  by  reviewing  the  survey  form  for  topics  to  pursue.  It  is  not  necessary  to  ask  all  of  the 

questions  that  follow,  nor  to  ask  them  in  precisely  the  wording  provided.  Dp  be  sure  to  follow  up  on  all  answers 

that  require  explanation  (Le.,  ask  "Why?” frequently). 

QUESTIONS  ABOUT  THE  PROGRAM  OVERALL 

1 .  How  do  you  feel  about  yoifr  choice  of  major  now  that  you’re  about  to  graduate? 

2.  What  did  you  expect  of  1|ie  ME  program  when  you  began  your  major? 

3.  What  do  you  think  about 'the  integration  of  the  program?  In  other  words,  do  you  see  how 
yotir  various  courses  fit  together?  Does  it  seem  like  a  “program”  or  merely  a  “sequence  of 
courses”? 

4.  WThat  do  you  think  about  the  current  structure  of  the  program?  (Including  the  number  and 
types  of  courses,  electives  vs.  requirements,  etc.) 

5.  [Refer  to  survey  page:]  If  you  have  taken  more  than  120  credits,  why  did  you  take  more  than 
the  required  number?  If  you  have  taken  just  120,  what  helped  you  to  finish  without 
additional  credits?  (Advising,  cafeteria-style  course  sampling,  change  majors,  double 
major?) 

6.  What,  if  any,  changes  in  your  undergrad  program  would  have  helped  better  prepare  you  for 
work  as  an  ME? 

7.  Do  you  think  the  ME  program  has  equipped  you  for  a  typical  day  on  the  job?  What  do  you 
think  a  typical  day  would  be  like?  What  did  you  learn  in  your  undergrad  program  that  will 
assist  you  in  your  T*  job? 

8.  Are  there  areas  of  your  undergraduate  program  that  you  feel  will  not  be  useful  to  you?  What 
are  they? 

QUESTIONS  ABOUT  TECHNICAL,  DESIGN,  AND  MANUFA  CTURING  SKILLS 

1 .  Do  you  feel  adequately  prepared  to  analyze  a  mechanical  engineering  problem?  Why  or  why 
not? 

2.  Do  you  feel  adequately  prepared  to  conduct  experiments?  Why  or  why  not? 
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3.  Do  you  feel  confident  about  your  computer  skills?  Why  or  why  not? 

4.  Given  your  coursework  in  manufacturing,  do  you  feel  adequately  prepared  to  enter 
manufacturing  (even  if  you  don’t  plan  to  do  so)?  Why  or  why  not? 

5.  To  what  extent  did  the  senior  design  course  prepare  you  for  work  as  a  mechanical  engineer? 
Which  aspects  of  the  course  were  most  and  least  useful  in  preparing  you  for  this  work? 

6.  If  you  completed  a  co-op,  did  you  find  it  valuable?  Why  or  why  not? 

QUESTIONS  ABOUT  NON-TECHNICAL  SKILLS 

1 .  Do  you  think  you’ve  gained  the  skills  necessary  to  be  an  effective  team  member  on  the  job? 
What  are  those  skills?  Was  there  a  particular  area  of  the  ME  program  that  helped  you  gain 
those  skills? 

2.  Do  you  think  you’ve  gained  the  skills  necessary  to  commxmicate  effectively  (orally  and  in 
writing)  on  the  job?  Which  parts  of  the  ME  program  helped  you  gain  those  skills? 

3.  Are  there  any  other  non-technical  skills  that  you  think  you’ll  need  on  the  job?  What  are 
they?  How  well  has  the  program  helped  you  gain  those  skills? 

QUESTIONS  ABOUT  TEACHING  AND  LEARNING  STYLES 

1 .  I’d  like  you  to  think  about  how  you  learned  in  your  last  two  years  in  the  program  (lecture, 
homework,  in  a  group,  alone,  hands-on,  by  applying  what  you  learned  to  real  life).  Oyerall, 
what  were  the  best  ways  for  you  to  leam?  What  helped  you  make  connections  with  the 
material?  Can  you  describe  an  example  of  an  assignment  or  project  that  you  found  especially 
effective  in  helping  you  leam? 

2.  Given  that  no  undergraduate  program  can  teach  you  everything  you’ll  ever  need  to  know  as  a 
mechanical  engineer,  do  you  feel  prepared  to  leam  on  your  own  after  graduation? 

3.  Where  have  you  received  the  most  useful  curricular  and  career  advice  and  information? 

What  types  of  information  have  been  most  useful  for  you? 

WRAP-UP  QUESTION 

1 .  Any  other  comments  on  the  progr^  or  “messages”  you’d  like  the  faculty  to  hear? 
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DEPARTMENT  of  MECHANICAL  ENGINEERING  ASSESSMENT 
ALUMNI  SURVEY  -  EXCERPT 


Career  Preparation: 

Please  rate  how  well  your  ME  undergraduate  education  at  UW-Madison  prepared  you  for  the 
following  activities  and  also  rate  how  frequently  you  engage  in  these  activities. 


Level  of  Preparation _ Frequency  of  Use 


Strorig 

Adequate 

Weak 

Daily 

Weekly 

Monthly 

Yearly 

Never 

Analyze  mechanical 
engineering  problems 

■ 

Design  &  conduct 
experiments 

/ 

/ 

Use  computers/software 

L  ’v 

Work  in  manufacturing 

Work  in  design 

Work  on  a  multi¬ 
disciplinary  team 

Communicate  orally 

Communicate  in  writing 

Make  technical  decisions 

Make  ethical  decisions 

Make  business  decisions 

Manage  projects 

Manage  people 

Consider  the  societal  or 
global  impact  of  my 
work. 

Teach  myself  new  things 

Overall,  how  well  prepared  do  you  believe  you  are  to  compete  within  your  field  or  current  area 
of  employment? 

O  Very  Prepared  O  Somewhat  Prepared  O  Somewhat  Unprepared  O  Very  Unprepared 


For  areas  in  which  you  feel  your  preparation  was  strong,  what  aspects  of  the  ME  program  gave 
you  that  preparation? 


For  areas  in  which  you  feel  your  preparation  was  weak,  what  aspects  of  the  ME  program  could 
be  improved  to  give  that  preparation? 


Were  there  any  areas  of  the  ME  undergraduate  program  that  you  felt  were  unnecessary?  If  so, 
please  identify  them  and  explain  why  they  seemed  superfluous. 
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